Search optimization technique for Domain Specific Parallel Crawler

Architectural framework of World Wide Web is used for accessing linked documents spread out over millions of machines all over the Internet. Web is a system that makes exchange of data on the internet easy and efficient. Due to the exponential growth of web, it has become a challenge to traverse all URLs in the web documents and handle these documents, so it is necessary to optimize the parallelize crawling process. In domain specific parallel crawler different domains are distributed among crawler for getting fast result. The crawler crawls the web periodically to maintain the freshness of repository but due to large amount of data, the relevant information does not update frequently. This paper proposes a novel technique that uses a Selection Factor algorithm for optimizing the search in Domain Specific Parallel Crawler and provide relevant information frequently in repository. Keywords— Search Engine, Parallel Crawler, Domain Specific Parallel Crawler, Selection Factor Algorithm, Page Rank